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REMARKS 

The Examiner is thanked for providing a careful review of this application. Claims 
1-2, 4-8, 10-12, and 15-21 have been amended. Claims 22-26 have been added. Claims 1- 
5 26 are pending after entry of the present Amendment. 

Information Disclosure Statement 

In accordance with the Examiner's request, a duplicate of the Information 
Disclosure Statement filed by the Applicant on October 3, 2000, will be sent via Examiner 
10 facsimile to 703-746-5850. 

Claim Objections 

Claim 21 was objected to because of an improper phrase "{entire?partial?}." Claim 
21 has been amended to remove the phrase "{entire?partial?}." 

15 Claims 13-17 were objected to as being dependent upon a rejected base claim. The 

Office has indicated that claims 13-17 would be allowable if rewritten in independent form 
including all of the limitations of the base claim and any intervening claims. Claim 22 has 
been added to represent claim 13 rewritten in independent form including all of the 
limitations of independent claim 12 from which it depends. Claim 23 representing claim 15 

20 has been added to depend from claim 22. Claims 24-25 representing claims 16-17, 
respectively, have been added to depend from claim 23. According to the Office's 
statement, claim 22 should be in condition for allowance. In following, claims 23-25 which 
depend from claim 22 should also be in condition for allowance. Additionally, claim 26 has 
been added to represent claim 14 rewritten in independent form including all of the 
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limitations of independent claim 12 from which it depends. According to the Office's 
statement, claim 26 should be in condition for allowance. 

As described below, claim 12 has been amended to be in better consideration for 
allowance. Therefore, original claims 13-17, which ultimately depend from claim 12, 
5 remain pending. 

Allowable Subject Matter 

The Applicants acknowledge that the Office has determined that claims 6-11 and 21 
are allowable. 

10 

Rejections under 35 U.S.C. § 102 

Claims 1 and 18-19 are rejected under 35 U.S.C. 102(e) as being anticipated by 
Norton et al. ("Norton") (U.S. Patent No.: 5,704,836). These rejections are respectfully 
traversed. 

15 With respect to claim 1 as amended, Norton does not teach "the input image 

processor programmed to (a) isolate the human form from the background in the captured 
video image by comparing the captured video image with a video image of the background 
without the human form." More specifically, Norton does not teach comparing a video 
image including a background and a human form with a video image including a 

20 background without a human form. 

Norton teaches the following, "Upon positioning the subject 28 within the field of 
view (block 106) and completing the sensitivity adjustment (block 108), the control system 
20 reads the subject frame 90 (block 110)." (column 14, lines 21-24) "The frame data 30 is 
then stored by the command unit 24 (block 112). When the frame data 30 for the initially 

25 read frame is stored, the control system 20 reads the subject frame 90 again (block 114)." 
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(column 14, lines 28-32) "For the sake of discussion herein, the previously read and stored 
frame will be called the previous frame and the subsequently read frame will be called the 
present frame. The command unit 24 now compares the present frame to the previous 
frame (block 116)," (column 14, lines 34-39) 



and a present frame, wherein both the previous and present frame include the subject or 
human form. Norton further teaches tracking the edge of the human form in both the 
previous and present frames to determine the direction and velocity associated with motion 
of the human form. In contrast to Norton, the presently claimed invention, requires that the 

10 human form be isolated from the background by comparing captured video images that 
contain both the background and the human form with captured video images that contain 
the background without the human form. 

With respect to claim 18 as amended, Norton does not teach "calculating arm 
position and movement data, wherein the calculating includes calculating angles of 

15 principle moment of the arm portions of the human form." More specifically, Norton does 
not teach calculating angles of principle moment of the arm portions. 

Norton teaches the following, "Once the edge of the subject 28 is determined, then 
the position of the subject 28 in the previous and the present frame provide a base from 
which the movement may be analyzed. The movement of the subject 28 as read by the 

20 command unit 24 is indicated by command vector C, from which velocity and direction are 
computed (blocks 124 and 126)." (column 15, lines 28-34) 

As indicated by the above-referenced teachings, Norton teaches calculating the 
velocity and direction of movement of the subject. However, Norton does not teach 
calculating angles of principle moment of the arm portions of the human form as required 

25 by the presently claimed invention. 



5 



As indicated by the above-referenced teachings, Norton teaches reading a previous 
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In view of the foregoing, the AppUcants respectfully submit that Norton fails to 
teach each and every element and limitation as required for a 35 U.S.C. 102 rejection. For 
at least these reasons, it is submitted that independent claims 1 and 18 are patentable over 
the cited art of record. For at least the same reasons, the Applicants respectfully submit that 
dependent claim 19 is patentable over the cited art of record. Therefore, the Office is 
respectfully requested to withdraw the 35 U.S.C. 102 rejections. 



Rejections under 35 U.S.C. § 103 

Claims 2-3 were rejected under 35 U.S.C. 103(a) as being unpatentable over Norton 
10 in view of Freeman (U.S. Patent No: 5,454,043). These rejections are respectfully 
traversed. 

Claims 2-3 depend from claim 1 as amended. For at least the reasons previously 
stated with regard to the rejections under 35 U.S.C. § 102, the Applicants submit that 
independent claim 1 as amended is patentable over the cited art of record. For at least the 

15 same reasons, the Applicants respectfully submit that dependent claims 2-3 are patentable 
over the cited art of record. 

Claims 4-5, 12, and 20 were rejected under 35 U.S.C. 103(a) as being unpatentable 
over Norton in view of Tsutsuguchi et al. ("Tsutsuguchi") (U.S. Patent No: 6,072,494). 
These rejections are respectfully traversed. 

20 With respect to claims 4-5, 12, and 20, neither Norton, Tsutsuguchi, nor any 

combination thereof teach or suggest all of the claim limitations. With respect to claim 4, 
neither Norton, Tsutsuguchi, nor any combination teach or suggest "capturing video images 
of a background without a human form and a background with a human form and 
processing those images to isolate the human form from the background." With respect to 

25 claim 5, neither Norton, Tsutsuguchi, nor any combination teach or suggest "capturing 
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video images of a background without a human form and a background with a human form 
with the device and using the image processor to process those images to isolate the human 
form from the background." With respect to claim 12, neither Norton, Tsutsuguchi, nor any 
combination teach or suggest "capturing video images with the device, the video images 
5 including an image of a background without a human form and an image of a background 
with a human form; using the image processor to process those images to isolate the human 
form from the background." With respect to claim 20, neither Norton, Tsutsuguchi, nor any 
combination teach or suggest instructions for "capturing video images with the device, the 
video images including an image of a background without a human form and an image of a 

10 background with a human form; using the image processor to process those images to 
isolate the human form from the background." 

As previously discussed with regard to the rejections under 35 U.S.C. § 102, 
Norton teaches the following, "Upon positioning the subject 28 within the field of view 
(block 106) and completing the sensitivity adjustment (block 108), the control system 20 

15 reads the subject frame 90 (block 110)." (column 14, lines 21-24) "The frame data 30 is 
then stored by the command unit 24 (block 112). When the frame data 30 for the initially 
read frame is stored, the control system 20 reads the subject frame 90 again (block 114)." 
(column 14, lines 28-32) "For the sake of discussion herein, the previously read and stored 
frame will be called the previous frame and the subsequently read frame will be called the 

20 present frame. The command unit 24 now compares the present frame to the previous 
frame (block 116)." (column 14, lines 34-39) 

As indicated by the above-referenced teachings, Norton teaches reading a previous 
and a present frame, wherein both the previous and present frame include the subject or 
human form. Norton further teaches tracking the edge of the human form in both the 

25 previous and present frames to determine the direction and velocity associated with motion 
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of the human form. In contrast to Norton, the presently claimed invention, requires that the 
human form be isolated from the background by comparing captured video images that 
contain both the background and the human form with captured video images that contain 
the background without the human form. 
5 The Office has relied upon Tsutsuguchi for teaching steps for calculating arm 

position and movement data in the broadest terms. The Office does not rely upon 
Tsutsuguchi for teaching isolation of a human form from a background. The Applicants 
respectfully submit that Tsutsuguchi, like Norton, fails to teach a requirement that the 
human form be isolated from the background by comparing captured video images that 

10 contain both the background and the human form with captured video images that contain 
the background without the human form. Therefore, neither Tsutsuguchi, Norton, nor any 
combination thereof teach or suggest isolating a human form from a background by 
comparing an image containing both the background and the human form to an image 
containing the background without the human form. 

15 In summary, to establish a prima facie case of obviousness, the references when 

combined must teach or suggest all the claim limitations. Therefore, it is respectfully 
submitted that the Office has not established a prima facie case of obviousness because the 
references when combined do not teach or suggest all of the claim limitations. For at least 
these reasons, the Applicants respectfully request that the rejection of independent claims 

20 4-5, 12, and 20, be withdrawn. For at least the same reasons, the Applicants respectfully 
submit that dependent claims 13-17 are patentable over the cited art of record. 

Additional Claim Amendments 



25 substance, various other amendments have been made to claims 1-2, 4-8, 10-12, and 15-21 



In addition to the amendments made to claims 1,4, 5, 12, 18, and 20 on matters of 
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to ensure clarity and proper use of antecedent basis. These various other amendments do 
not affect the meaning or scope of the claims as previously worded. 



The Applicants respectfully submit that all of the pending claims are in condition 
5 for allowance, A notice of allowance is respectfully requested. If the Examiner has any 
questions concerning the present amendment, the Examiner is kindly requested to contact 
the undersigned at (408) 749-6900 ext. 6903. If any additional fees are due in connection 
with filing this amendment, the Commissioner is also authorized to charge Deposit 
Account No. 50-0805 (Order No. SONYP003). A duplicate copy of the transmittal is 
10 enclosed for this purpose. 



15 



Respectfully submitted, 
Martine & PENILLA, L,L,P, 




*^lbertS.Penilla, Esq. 
Reg. No. 39,487 



20 Martine & Penilla, LLP 

710 Lakeway Drive, Suite 170 
Sunnyvale, California 94086 
Tel: (408)749-6900 
Customer Number 25,920 
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IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 



In re application of: 

Marks, Richard 

Application No: 09/654,141 

Filed: September 1 , 2000 

for: user input device and method for 
Interaction with Graphic Images 



Docket No: SONYP003 

Group Art Unit: 2674 
Examiner: Nguyen, Francis N. 

Date: February 26, 2003 



MARKED UP AMENDMENT 



Marked Up Claims 



1, (Amended) An input device for providing a signal to effect one of 
translational movement, rotational movement, and both translational [and/or] and 
rotational movements of an object on a graphical display, comprising: 

a device for capturing video images; 

an input image processor that translates captured video images of human arm 
motion into signals that are delivered to an output image processor, the input image 
processor programmed to (a) isolate [the] a human form from [the] a background in [the] a 
captured video image by comparing the captured video image with a video image of the 
background without the human form ; (b) determine [the] a position and a movement of 
arms of the human form [arms]; and (c) generate an output signal responsive to one of the 
position, the movement, and both the position [and/or] and the movement of the human 
arms; and 
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an output image processor that is programmed to effect one of translational 
movement, rotational movement, and both translational [and/or] and rotational movement 
of an object on a graphical display in response to the signals received from the input image 
processor. 

2. (Amended) The input device of claim 1 wherein the output image processor 
changes the graphical display according to [the] a perspective of what a flying object would 
see. 

4. (Amended) A method for generating signals to effect one of translational 
movement, rotational movement, and both translational [and/or] and rotational movements 
of an object on a graphical display using human arm position and movement data, 
comprising: 

providing an image processor and a device for capturing video images; 

capturing video images of a background without a human form and a background 
with a human form: [and] 

processing [those] the captured video images to isolate [a]tjTe human form from 
[a] the background; 

isolating [the] arm portions of the human form; 

calculating [the] arm position and movement data; and 

generating a signal responsive to the arm position and movement data for effecting 
one of translational movement, rotational movement, and both translational [and/or] and 
rotational [movement] movements of an object on a graphical display. 
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5. (Amended) A method for generating signals using one of human arm 
position data, human arm movement data, and both human arm position [and/or] and 
movement data, comprising: 

providing an image processor and a device for capturing video images; 

capturing video images of a background without a human form and a background 
with a human form with the devicei [and] 

using the image processor to process [those] the captured video images to isolate 
falthe human form from [a]the background; 

isolating [the] arm portions of the human form from a captured video image using 
the image processor; 

calculating [the] arm position and movement data using the image processor; and 

generating a signal responsive to the arm position and movement data using the 
image processor. 

6. (Amended) A method for generating signals to effect one of translational 
movement, rotational movement, and both translational [and/or] and rotational movements 
of an object on a graphical display using one of human arm position data, human arm 
movement data, and both human arm position [and/or] and movement data, comprising: 

providing an image processor and a device for capturing a video sequence; 
capturing, from the video sequence, a frame that does not include at person; 
isolating a view comprising a foreground subject image view by performing an 
algorithm on the video sequence and the frame that does not include the person; 

determining whether the isolated view includes [the] an image of a person; 
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determining [the] a horizontal extent of [the subject's] a torso in the image of the 
person so as to isolate [the] arm portions of the [human form] person in [each] frames of 
the captured video [frame] sequence ; 

computing [the] arm angles by calculating angles of principle moment of [the] 
nonzero pixels in the arm portions of the [video] image of the person ; and 

generating an arm position data signal responsive to arm angles for effecting one of 
translational movement, rotational movement, and both [the] translational [and/or] and 
rotational movement of an object on a graphical display. 

7. (Amended) The method of claim 6 wherein the step of determining whether 
the isolated view includes the image of the [a] person comprises the steps of: 

counting [the] a total number of nonzero pixels in the foreground image; 
ensuring that the total number of nonzero pixels in the foreground image falls 
within a range defined by a minimum and a maximum threshold number of pixels. 

8. (Amended) The method of claim 6 wherein the algorithm in the isolating 
step involves subtracting the frame that does not include [a] the person from [the] 
individual frames in the video sequence. 

10. (Amended) The method of claim 6 wherein the arm [position/movement] 
position data [signals] signal generated in the generating step [are] is selected from the 
group consisting of signals related to object airspeed acceleration, bank angle, and pitch 
angle. 
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11. (Amended) The method of claims 6 wherein the ami [position/movement] 
position data [signals] signal generated in the generating step [are] is determined with the 
inclusion of smoothing constants. 

5 12. (Amended) A method for generating signals for use in a flight simulator 

graphical display using human arm position data to effect one of translational movement, 
rotational movement, and both translational [and/or] and rotational movement, comprising: 
providing a device for capturing video images and an image processor; 
capturing video images with the device , the video images including an image of a 
10 background without a human form and an image of a background with a human form; 
[and] 

using the image processor to process [those] the captured video images to isolate 
[a] the human form from [a]the background; 

isolating [the] arm portions of the human form from a captured video image using 
15 the image processor; 

calculating arm position and movement data using the image processor; and 

generating a signal responsive to the arm position and movement data using the 
image processor for use in generating [the] a state of a flight simulator graphical display. 

20 15. (Amended) The method of claim 13 further including [the] a step of 

generating flapping noises corresponding to [the] movement of the wings of the flying 
creature. 

16. (Amended) The method of claim 15 wherein [the] a volume of the flapping 
25 noises increases with an increased rate of [captured] arm motion. 
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17. (Amended) The method of claims 15 wherein the flapping [noise is] noises 
are triggered when [the] a signed time rate of change of [the] an average of [the] arm 
angles exceeds a pre-determined threshold. 

18. (Amended) An article of manufacture embodying a program of instructions 
executable by a machine, the program of instructions including instructions for: 

capturing video images^ [and] 

processing [those] the captured video images to isolate a human form from a 
background; 

isolating [the] arm portions of the human form; 

calculating [the] arm position and movement data , wherein the calculating includes 
calculating angles of principle moment of the arm portions of the human form ; and 

generating a signal responsive to the arm position and movement data for effecting 
one of translational movement, rotational movement, and both translational [and/or] and 
rotational movement of an object on a graphical display. 

19. (Amended) The article of manufacture of claim 18 wherein the signal 
generated by the program of instructions is used to generate [the] a state of a flight 
simulator graphical display. 

20. (Amended) An article of manufacture embodying a program of instructions 
executable by a machine, the program of instructions including instructions for: 
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capturing video images with [the] a device , the video images including an image of 
a background vyithout a human form and an image of a background v^ith a human form; 
[and] 

using [the] an image processor to process [those] the captured video images to 
5 isolate [a]the human form from [a]the background; 

isolating [the] arm portions of the human form from a captured video image using 
the image processor; 

calculating [the] arm position and movement data using the image processor; and 
generating a signal responsive to the arm position and movement data using the 
10 image processor. 



21. (Amended) An article of manufacture embodying a program of instructions 
executable by a machine, the program of instructions including instructions for: 

capturing, from [the] a video sequence, a frame that does not include a person; 
15 isolating a view [(foreground/subject image)] of an image of the person by 

performing an algorithm on the video sequence and the frame that does not include [a] the 
person; [{entire? partial?}] 

determining whether the isolated view includes the image of [a] the person; 
determining [the] a horizontal extent of [the subject's] a torso of the person in the 
20 image so as to isolate [the] arm portions of the [human form in a/the/each captured video 
frame] person in the image ; 

computing [the] arm angles by calculating angles of principle moment of [the] 
nonzero pixels in the arm portions of the [video] person in the image; and 

generating one of an arm position data signal, an arm movement data signal, and 
25 both [an arm position/movement] arm position and movement data [signal] signals 
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responsive to the computed arm angles for effecting one of translational movement, 
rotational movement, and both [the] translational [and/or] and rotational movement of an 
object on a graphical display. 



5 22. (New) A method for generating signals for use in a flight simulator 

graphical display using human arm position data to effect one of translational movement, 
rotational movement, and both translational and rotational movement, wherein the flight 
simulator graphical display includes as an object a flying creature that moves wings, 
comprising: 

10 providing a device for capturing video images and an image processor: 

capturing video images with the device; 

using the image processor to process the captured video images to isolate a human 
form from a background; 

isolating arm portions of the human form from a captured video image using the 
15 image processor; 

calculating arm position and movement data using the image processor; and 

generating a signal responsive to the arm position and movement data using the 
image processor, the signal to be used in generating a state of the flight simulator graphical 
display. 

20 

23. (New) The method of claim 22 further including a step of generating 
flapping noises corresponding to a movement of the wings of the flying creature. 



24. (New) The method of claim 23 wherein a volume of the flapping noises 
25 increases with an increased rate of arm motion. 
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25. (New) The method of claim 23 wherein the flapping noises are triggered 
when a signed time rate of change of an average of calculated arm angles exceeds a pre- 
determined threshold. 

5 

26. (New) A method for generating signals for use in a flight simulator 
graphical display using human arm position data to effect one of translational movement, 
rotational movement, and both translational and rotational movement, comprising: 

providing a device for capturing video images and an image processor: 
10 capturing video images with the device; 

using the image processor to process the captured video images to isolate a human 
form from a background; 

isolating arm portions of the human form from a captured video image using the 
image processor; 

15 calculating arm position and movement data using the image processor; and 

generating a signal responsive to the arm position and movement data using the 
image processor, the signal to be used in generating a state of the flight simulator graphical 
display, wherein the flight simulator graphical displav depicts a change in perspective of 
what a flying creature would see. 

20 
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